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Abstract 

The goal of this study was to explore how students debate with their peers within a designed context using a 
digital dialogue game, and whether their epistemic beliefs are significant to the outcomes. Epistemic beliefs 
are known to colour student interactions within argumentative discourse, leading some students to hold back 
from interactions. By designing an online small group activity based around an issue both important and 
controversial to the students, with multiple viewpoints in each group and with the scaffolding provided by a 
dialogue game, it was examined whether these epistemic effects were still evident within their argumentative 
discourse. Furthermore, the study examined whether the activity design improves students' willingness to 
argue with each other, and their openness to attitudinal change. A pretest, posttest design was used with 
students who were assigned to groups of four or five and asked to argue on a controversial topic. Their aim 
was to explore various perspectives and to debate the pros and cons of the use of Genetically Modified 
Organisms (GMOs). While previous research has shown that some epistemic beliefs lead to less critical 
engagement with peers, the results presented here demonstrate that activity design is also an important 
factor in successful engagement within argumentative discourse. 
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Introduction 

The advancements of the educational and communication technologies have acted as a catalyst for 
educational transformation in the past two decades. With the growth of the Internet and related 
technologies and wide accessibility to the World Wide Web, there is an inevitable need in modern 
society to confront and analyse complex issues. To prepare society for tackling these complexities, 
professionals and experts from diverse disciplines need to collaborate in new learning and working 
contexts. It is the case in education particularly, that students need to gain ample experience working 
in learning groups to become capable and qualified professionals who can actively participate in the 
knowledge society, analyse, synthesize, and cope with complex and societal issues. To do so, students 
need to be able to engage in dialogical argumentation, build arguments and support a position, to 
consider and weigh arguments and counter-arguments, to test, enlighten, and clarify uncertainties, 
and thus review misconceptions and correct false viewpoints (Aleixandre-Jimenez, 2007). 

In line with the innovation and latest developments in the field of educational technology, many 
international universities have started to develop open and distance learning programmes in addition 
to their on-campus programmes. International professionals and students are keenly interested in 
such programmes due to the possibility to combine work, family responsibilities, and study; the lower 
annual costs; and the assumed flexibility. Like any other programme, open and distance learning 
programmes may have their own specific risks and disadvantages, especially with respect to lack of 
engagement and high student dropout rates. Despite the benefits of open and distance learning 
programmes, the physical separation of learners from their peers and instructors may cause a weaker 
sense of belonging to a classroom community and a lack of engagement, communication, and 
interactivity (Chaiprasurt & Esichaikul, 2013). Many scholars have concluded that this lack of 
belonging to a classroom community may reduce learners’ motivation which could also result in 
dissatisfaction, poor performance, and dropout (Balaban-Sali, 2008; Rau, Gao, & Wu, 2008). 
Therefore, additional motivational factors are needed to compensate for the physical separation of 
students from their peers and also their instructors in open and distance learning programmes. 

One approach for stimulating learners’ motivation and satisfaction with open and distance learning 
programmes is to provide them with the opportunity to work together as a learning group to 
encouraging a spirit of social learning, praising and recognition of one another’s positive efforts, and 
fostering knowledge about the learning process (Sharan & Shaulov, 1990). The collaborative learning 
can thus be seen as a motivational factor for students to work on their assignments in a challenging 
yet pleasant learning group rather than individual work which in turn may positively influence 
students’ engagement and academic, social, and attitudinal outcomes (Miyake, 2007). 

Argumentation is a vehicle for collaborative learning process not only for traditional forms of 
classrooms but also for blended and online learning settings such as open and distance learning 
programmes in higher education (see Noroozi, Busstra, Mulder, Biemans, Tobi, Geelen, et al., 2012). 
Argumentation is considered to be significant to education due to the importance of discourse in the 
acquisition of scientific knowledge (Osborne, 2010). Due to the importance of argumentation, 
scientific evidence suggests to explicitly design for critical discussion, argumentation, and reasoning 
rather than expecting it to occur naturally in open and distance learning programmes (see Noroozi, 
Weinberger, Biemans, Mulder, & Chizari, 2012; Ravenscroft, 2011). 
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Engaging in dialogical argumentation with learning partners for carrying out complex learning tasks 
can be more difficult in traditional settings rather than blended and online learning environments. 
There could be several reasons for the difficulties for argumentation in traditional (face-to-face) 
settings as compared with blended and online learning environments. Traditional class settings may 
create some emotional and social barriers for students to engage in face-to-face argumentation with 
learning partners. For example, there are some students who might be holding epistemic emotions 
(being curious and anxious when receiving counter-arguments in a face-to-face situation), while 
others might be emotional based on their achievements (being proud of their success or shameful on 
their failure) during argumentation-based collaborative learning. This might just hold back some 
students to engage in dialogical argumentations with their learning peers in the face-to-face setting 
because they want to avoid getting into a verbal fight with their learning partner(s) (Andriessen, 
2006). These difficulties for engaging in dialogical argumentation can to some extent be tackled in 
online or blended learning environments when students’ identities are not revealed to their learning 
partners. Being unknown to learning partners through nicknames in online and blended learning 
environments may provide a safer and more respectful learning opportunity for learning groups who 
engage in argumentative discourse activities. Even when the unanimity of the learning partners is kept 
secret, students may still face difficulties during collaborative argumentation-based learning. The ill- 
defined nature of argumentation makes it almost impossible for learners to follow a set of strict rules 
and unbending laws on constructing arguments, responding to counter-arguments, and engaging in 
transactive argumentation in order to gain and construct knowledge, reject false viewpoints, refine 
and modify claims, and eliminate misunderstandings and misconceptions about the issue at stake (see 
Andriessen, 2006; Noroozi, Weinberger, et al., 2012). 

Various instructional approaches have been proposed to help students learn the skills of 
argumentation and critical discussion for engaging in effective dialogical argumentation. The most 
recent approach is the use of computer-support, such as online argument awareness representations, 
computer-supported collaboration scripts, and digital dialogue games to scaffold collaborative 
argumentation and support the building, representing, and sharing of arguments with the aim of 
learning. These computer-support systems are fully described in studies by Kirschner, Buckingham- 
Shum, and Carr (2003), McLaren, Scheuer, and Miksatko (2010), Scheuer, Loll, Pinkwart, and 
McLaren (2010), as well as Noroozi, Weinberger, et al. (2012). Most of these computer-support 
systems lack motivational aspects of learning leading to less student satisfaction and motivation in 
these instructional environments. Specifically, they focused mostly on challenging intellectual tasks, 
prompting and supporting students in arguing and debating issues in science (Suthers, 2003), the law 
(Pinkwart, Ashley, Lynch, & Aleven, 2009), and ethics (McLaren et al., 2010), without regard for the 
students’ motivation. This is a striking omission since in real educational settings, motivational factors 
and willingness to argue play a key role in the extent to which students approach or avoid arguments 
(Infante & Rancer, 1982). Willingness to argue can be influenced by the learner's level of 
assertiveness, which may in turn determine whether they engage in, or avoid, critical reasoning and 
arguments (Nussbaum & Bendixen, 2003). Due to the fear of getting into verbal fight, some learners 
may avoid disagreements with their learning partners, while other students may dislike to be 
questioned by their peers (Nussbaum, Hartley, Sinatra, Reynolds, & Bendixen, 2004). Also, less 
assertive students avoid arguments due to the challenging and competitive nature of argumentation 
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and debating (Nussbaum, Sinatra, & Poliquin, 2008). For all these students, the structuring of 
discussion within a game context may make a significant difference to their motivations and level of 
interactions with peers. 

Another factor that is crucial for the extent to which students engage in, or avoid, critical reasoning 
and arguments is their epistemic beliefs (Noroozi, Weinberger, et al., 2012; Nussbaum et al. 2008). 
Epistemic beliefs can be defined as one’s own opinion on the nature, structure, and certainty of 
knowledge and justification for knowing with regard to knowledge acquisition (see Hofer, 2000; 
Hofer & Pintrich, 1997; King & Kitchener, 1994). From this perspective, students’ epistemic beliefs can 
be labelled as: (a) absolutism, (b) multiplism, and (c) evaluativism. Absolutists view knowledge as 
objective, simple, certain, and fixed that cannot be changed meaning that there is only one right or 
wrong answer and only authority figures have those answers. Multiplists perceive knowledge as 
subjective and contextual where viewpoints are seen as mere opinions. In this case, students are 
exposed to various perspectives of the issue at hand with the aim of concluding that one point of view 
is as good as another. In the most developed and sophisticated scenario, evaluativists perceive 
knowledge as verified true belief meaning that there are multiple possibilities in which knowledge 
claims must always be evaluated for their quality of arguments in different contexts (see Muis, 2007). 
Scientific empirical evidence has shown that students engage in argumentation differently with 
respect to their epistemic beliefs. For example, multiplists are less critical regarding inconsistencies 
and misconceptions and less interactive with their partners than other belief groups (Nussbaum et al. 
2008). It is also shown that evaluativists are more critical and active in eliciting information from 
their partners (Nussbaum et al. 2008), compared with absolutists who are less inclined to explore 
alternative solutions (Oh & Jonassen, 2006). 

Epistemic beliefs can therefore be seen as a factor that influences the way students engage in 
argumentative discourse and critical discussion and reasoning. With regard to willingness to argue, 
since argumentation and debating involve social learning processes (O’Keefe, 1982) and guide student 
attention towards exploring various sides of issue at stake (Noroozi, Biemans, Busstra, Mulder, & 
Chizari, 2011; Nussbuam et al., 2008), we hypothesise that students’ attitudes towards GMOs would 
be modified after the discourse. The goal of this study is to explore how undergraduate students with 
various epistemic beliefs engage in argumentative discourse by exposing them to a controversial topic 
and conflicting views within a dialogue game which is fun to play, but encourages challenges. Prior 
research has not investigated the effects of epistemic beliefs on student argumentative discourse when 
student willingness to argue is enhanced by such activity design. The picture is also unclear when it 
comes to relationship between argumentative discourse activities, epistemic beliefs, and change in 
attitude. Therefore, the main question for this study is the following: what is the relationship between 
students’ argumentative discourse activities, epistemic beliefs, and change in attitude when exposing 
them to an activity design with enhanced motivational characteristics? 
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Method 


Context and Participants 

The study took place at Wageningen University in the Netherlands, with a focus on the life sciences, 
especially food and health, sustainability, and the healthy living environment. Students at this 
university are encouraged to combine natural and social sciences: from plant sciences to economics 
and from food technology to sociology. The participants were 25 BSc students who enrolled for the 
168-h course “Life-Science Communication and Learning in the Digital Age.” This new course is part 
of the Minor program on Environmental Education, which consists of Applied Environmental 
Education and Communication; Sustainable Development: Integrating Worldviews, Disciplines and 
Practices; Life-Science Communication and Learning in the Digital Age; and Didactic Skills courses 
with a total of 24 credits. In this course, students acquire insights into the possibilities and limitations 
of communication and learning in the digital age particularly in a life-sciences and sustainability- 
oriented context. Students explore how their own engagement in screen-based interaction affects their 
own ability to communicate and the way they learn. The mean age of the participants was 22.00 (SD = 
1.82), and the majority (80%) were female. This almost mirrors the proportion of female and male 
students at Wageningen University. 

Participants were divided into five groups of four students, and one group of five students, taking into 
account their perspective on the controversial issue of the discussion. This was done to stimulate the 
discussion in the group by making sure that each group includes students who are neutral, in favour 
of, and against the controversial issue. Although all participants were distributed around a single 
classroom and even though they knew one another in advance, nicknames were assigned to them so 
that they did not know who other members of their online group were. 

Materials 

The topic for discussion was the application of Genetically Modified Organisms (GMOs) in 
agriculture. Students were provided with the description of the controversial issue and a summary of 
the theoretical text regarding GMOs. They were also provided with some additional links to websites 
to further study the concept of the GMOs. For example, in the description of the controversial issue, it 
was explained that: 

GMOs have been touted as both nightmare and salvation—a nightmare to human health and 
biodiversity, and salvation for human food production, reducing pesticide emissions to the 
environment, and, by extension, biodiversity and human health. In short—the application of 
GMOs in our global food chains is rife with controversy. And despite the seeming ease with 
which one can paint the controversial picture of GMOs, the matter itself is rather complex. 
The financial and corporate stakes are sky high and of importance to our economies. Even 
issues with intellectual property rights on certain genes, as well as issues with plant breeders’ 
rights have come to the fore. In sum, the application of GMOs should be something that we 
can argue about—at length! 


212 



Impacts of a Digital Dialogue Game and Epistemic Beliefs on Argumentative Discourse and Willingness to Argue 

Noroozi, McAlister, and Mulder 


The students’ task was to read materials, discuss, and argue the GMO issue with three other members 
in the group while taking into account the various perspectives on the need—or lack thereof—of 
implementing GMOs in agriculture. 

Learning Environment: A Digital Dialogue Game 

The four or five learning partners in each group were distributed over different locations of a 
classroom with anonymous online names. A synchronous text-based discussion board called InterLoc 
was used in the study for the collaboration phase. InterLoc is a Web-enabled platform for structured 
dialogue and discussion that is used to stimulate discussions between members of a group in an active 
learning environment by getting students thinking together about topics, media, or material that is 
relevant to them. It guides the interaction style for synchronous dialogue—promoting reasoning, 
critical discussion, and justified arguments—and allows the players to produce reusable content from 
their group experiences using a saved transcript. This guiding is done by requiring a choice of a 
sentence opener to begin each contribution to the discussion. A variety of sentence openers 1 are 
embedded in the design for provoking and promoting students’ reasoning and the argumentative 
dialogue processes and practices of the players (see Figure 1 for a screenshot of the game 
environment). For instance, “I agree because...” encourages a player to provide a fully reasoned 
agreement. Other sentence openers deal with statements, evidence, support or criticism, and 
conclusions. In all cases these are formulated to cue the players to provide fuller, constructive 
contributions, which require more forethought. 

Equally important for guiding, and a key feature of the game, is the list of suggested openers for 
players' responses to others, dynamically based on what has gone before. The suggested list of openers 
for replies is derived from a conception of how a well-reasoned discussion should proceed, (i.e., from 
statements to fuller explanations) and from evidence to deriving justified conclusions. Each opener in 
the suggested list therefore explores more deeply the current stage or takes the discussion to the next 
stage 2 3 . 

The epithet of a dialogue game accurately describes the interactions within the discussion, as in a 
game there are rules about what (dialogue) moves can be made at different times, and so it is with 
InterLoc*. The presentation of an InterLoc discussion as a game to students is important 
psychologically because of the motivational aspects of participating in games; being involved in a 
game allows participants full reign to be critical, because such moves are sanctioned within a game 
environment. 

The full description of the game and the roles of the players and tutors can be found in in the InterLoc 
literature which examines the differences in group discussions through using InterLoc versus using a 


1 There are 53 sentence openers in the default game which have been carefully chosen and refined in hundreds 
of trials to ensure both usability for players and guidance towards the desired style of dialogue. 

2 However the suggested list can be overridden by the player if they require to make a specific contribution 
and they can choose from the full set of openers. 

3 In fact, within InterLoc there are alternative games the administrator can select-the most commonly used is 
the (default) Critical Reasoning game. 
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free discussion board (for example McAlister, Ravenscroft, & Scanlon, 2004; Ravenscroft, 2011; 
Ravenscroft & McAlister, 2006). 



Figure 1. A screenshot of the digital dialogue game with associated sentence opener. 

Procedure 

Overall, the session took about 2.5 hours and consisted of three main phases: 

1. During the introduction and pretest phase, which took 20 minutes, students first received 

introductory verbal explanations for about five minutes on the purpose of the game. Next, 
they were asked to complete several questionnaires (15 minutes) on demographic 
variables, their preliminary opinion on the GMO issue, as well as their epistemic beliefs. 

2. In the test-phase, the game began and lasted for about 70 minutes. Specifically, students were 

first oriented to the InterLoc learning environment and acquainted with its functionalities 
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followed by a short hands-on training exercise (10 minutes). Subsequently, they were 
asked to discuss and argue their positions on the various aspects of the GMO issue with 
their learning partners (60 minutes), followed by a 10 minute break. 

3. During the posttest and debriefing phase (60 minutes), students were asked to fill out several 
questionnaires to assess their satisfaction with the learning experience and its outcomes 
(15 minutes). Finally, there was a plenary verbal session in which students expressed and 
shared their opinions on their learning experience using the digital dialogue game with 
their fellow classmates and also the teacher and the researcher (40 minutes). 

Measurement of Students' Attitudinal Change 

A pretest posttest questionnaire was used to measure students’ attitudinal change on the GMO issue. 
This questionnaire consisted of two questions on a 5-point Likert scale ranging from strongly 
disagree, disagree, neutral, agree, through to strongly agree. Specifically, both in the pretest and 
posttest, each student was asked to indicate the extent to which they agreed with the following two 
statements: (a) GMOs should be further developed for the market to improve sustainability; and (b) 
GMOs are a danger to biodiversity. The data from the pretest served as an indicator of students’ 
preliminary opinions on the GMO issue in order to assign learning groups with the full range of 
opinions. Then, the data from posttest were compared with the pretest data in order to detect any shift 
of the student opinion on the GMO from pretest to posttest. 

Measurement of Epistemic Beliefs 

The students were measured on their epistemic beliefs using a 15-item instrument developed by Kuhn 
Cheney, and Weinstock (2000). This instrument was used to measure students’ beliefs about 
knowledge according to the judgement domains. Each item of this questionnaire consisted of a pair of 
contrasting statements, attributed to two individuals—Robin and Chris—in five domains including (a) 
judgements of taste, (b) aesthetic judgements, (c) value judgements, (d) judgements of truth about the 
physical world, and (e) judgements of truth about the social world. Following each pair of statements, 
this question was posed: 

Can only one of their views be right, or could both have some rightness? 

Response options were as follows: 
only one right (coded as absolutist) 
both could have some rightness 

The following question, which was contingent on the response to the first, was as follows: 

Could one view be better or more right than the other (If both could have some rightness was 
selected)? 

one could not be more right (coded as multiplist) 
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one could be more right than the other (coded as evaluativist) 

Based on the data from this questionnaire, each student was classified into three epistemic 
orientations: absolutists, multiplists, and evaluativists (Kuhn et al., 2000; Nussbaum et al., 2008). An 
absolutist believes that only one answer could be right. A multiplist believes that all opinions can be 
equally valid, so one opinion could not be more right than another in their view. An evaluativist 
believes that criteria exist whereby opinions or judgements can be evaluated and one can be shown to 
be better than another (so one opinion or judgement could be more right than another). Therefore, 
each student was classified as an absolutist when the first response option (i.e., only one right ) was 
dominant in his or her answers to the questions in various judgement domains. Accordingly, each 
student was classified as a multiplist when the second response option (i.e., both could have some 
rightness but one could not be more right ) was dominant in his/her answers to the questions in 
various judgement domains. And finally, each student was classified as an Evaluativist when the third 
response option (i.e. both could have some rightness but one could be more right than the other ) was 
dominant in his or her answers to the questions in various judgement domains. The reliability of the 
epistemic belief measure was satisfactory (Cronbach a = .74). 

Argumentative Discourse Activities 

A content analysis coding scheme was adapted to measure quality of argumentative discourse 
activities. This was done to assess how well the discussion notes of each student and also his or her 
group reflected the criteria of a scientific argument (Noroozi, Weinberger, Biemans, Mulder, & 
Chizari, 2013; Noroozi, Teasley, Biemans, Weinberger, & Mulder, 2013; Noroozi, Biemans, 
Weinberger, Mulder, & Chizari, 2013; Weinberger & Fischer, 2006). Every message posted during the 
online discussion on the GMO issue was coded as one of the following: externalization, elicitation 
(asking a question), agreement, integration, or disagreement. 

When students displayed a claim, the message was coded as externalization. Typically, this was the 
first message in a thread, though sometimes students might juxtapose externalizations (i.e., reply to 
earlier externalizations by a further externalization). It was also distinguished if the externalization 
was supported with evidence or not. When students asked for, or invited a reaction from their learning 
partners, we coded the message as elicitation and the answering response as externalisation. When 
students agreed with what had been said before, without any modification, or just by repeating what 
was said, the message was coded as agreement. Students might have taken over perspectives from 
their peers and built syntheses of (various) arguments and counter-arguments that learning partners 
had uttered before. In such a case, the message was coded as integration. Also, small additions to a 
peer’s utterances were coded as integration. Any rejection, denial, or negative answer or evaluation 
was coded as disagreement. Beyond saying “No” or “I disagree”, any kind of modification or 
replacement of what had been said before was also coded as conflict. Thus, small repairs to a peer’s 
utterances were coded as conflict. This included taking note of the phenomenon of alleviating 
critiques by initializing responses with phrases such as “I totally agree, but....” 
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Results 

Table l presents descriptive information on students’ epistemic orientation (based on Kuhn et al., 
2000), including attitudinal change, number of discussion messages per individuals and groups, and 
type of contribution based on argumentative discourse activities. This shows that 16 (64%) of the 
participants were classified as multiplist, 9 (36%) as evaluativists and none as absolutists. There were 
a total of 403 discussion messages generated during the discourse, with an average of 16.12 per 
student (SD = 5.227). Out of 403 discussion messages, 129 messages were categorized as 
externalization, 83 messages as elicitation, 68 messages as agreement, 66 messages as integration, 
and 57 messages as disagreement. In total, Group 1 produced 61 messages, Groups 2 and 3 each 
produced 69 messages, Group 4 produced 90 messages; Group 5 produced 49 messages and Group 6 
produced 67 messages. 
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Table 1 


Students’ Epistemic Orientation, Attitude Shift, Number of Messages, and Types of Contribution 


Learning 

group 

Nick¬ 

name 

Epistemic 

orientation 

Attitude 

shift 

Number 

of 

messages 

Message 
proportion 
in group 
( 96 ) 

Externalizat 

ion 

N % 

Types of contribution 

Elicitation Agreemen Integratio 
t n 

N % N % N % 

Disagreeme 

nt 

N % 

Group l 

Total 

Simon 

Multiplist 

Yes 

13 

21.31 

3 

23.1 

2 

15-4 

2 

15-4 

2 

15-4 

4 

30.8 

messages: 

61 

Robert 

Multiplist 

No 

21 

34-43 

4 

19.0 

2 

9-5 

3 

14-3 

8 

38.1 

4 

19.0 


Jane 

Evaluativist 

Yes 

15 

24-59 

3 

20.0 

4 

26.7 

1 

6.7 

5 

33-3 

2 

13-3 


Tom 

Multiplist 

No 

12 

19.67 

4 

33-3 

3 

25.0 

2 

16.7 

1 

8-3 

2 

16.7 

Group 2 

Total 

Carla 

Multiplist 

Yes 

17 

25.00 

7 

41.2 

4 

23-5 

2 

11.8 

3 

17.6 

1 

5-9 

messages: 

68 

Anke 

Evaluativist 

No 

21 

30.88 

8 

38.1 

4 

19.0 

6 

28.6 

2 

9-5 

1 

4-8 


Eline 

Multiplist 

No 

12 

17-65 

5 

41.7 

1 

8-3 

3 

25.0 

1 

8-3 

2 

16.7 


Femke 

Multiplist 

No 

l8 

26.47 

4 

22.2 

5 

27.8 

2 

11.1 

4 

22.2 

3 

16.7 

Group 3 

Total 

Anna 

Evaluativist 

Yes 

26 

38.24 

9 

34-6 

2 

7-7 

1 

3-8 

6 

23.1 

8 

30.8 

messages: 

68 

Martin 

Multiplist 

Yes 

10 

14.70 

5 

50.0 

1 

10.0 

0 

0.0 

0 

0.0 

4 

40.0 


Sara 

Evaluativist 

Yes 

15 

22.06 

8 

53-3 

3 

20.0 

2 

13-3 

1 

6.7 

1 

6-7 
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Learning 

group 

Nick¬ 

name 

Epistemic 

orientation 

Attitude 

shift 

Number 

of 

Message 

proportion 




Types of contribution 








messages 

in group 
(%) 

Externalizat 

ion 

Elicitation 

Agreemen 

t 

Integratio 

n 

Disagreeme 

nt 







N 

% 

N 

% 

N 

% 

N 

% 

N 

% 


Ester 

Evaluativist 

No 

17 

25.00 

8 

47.1 

1 

5-9 

0 

0.0 

0 

0.0 

8 

47.1 

Group 4 

Total 

Patricia 

Multiplist 

Yes 

21 

23-33 

5 

23.8 

11 

52.4 

4 

19.0 

0 

0.0 

1 

4-8 

messages: 

90 

Bianca 

Multiplist 

Yes 

9 

10.00 

1 

ll.l 

1 

11.1 

4 

44-4 

3 

33-3 

0 

0.0 


Ilona 

Multiplist 

No 

20 

22.22 

6 

30.0 

2 

10.0 

8 

40.0 

2 

10.0 

2 

10.0 


Eelco 

Multiplist 

Yes 

17 

18.89 

6 

35-3 

4 

23-5 

1 

5-9 

4 

23-5 

2 

11.8 


Jeroen 

Evaluativist 

No 

23 

25-56 

2 

8.7 

9 

39-1 

6 

26.1 

5 

21.7 

1 

4-3 

Group 5 

Total 

Hannek 

e 

Multiplist 

No 

10 

20.41 

6 

60.0 

1 

10.0 

0 

0.0 

2 

20.0 

1 

10.0 

messages: 

49 

Amanda 

Multiplist 

No 

7 

14.29 

4 

57-1 

0 

0.0 

3 

42.9 

0 

0.0 

0 

0.0 


Elizabet 

h 

Multiplist 

Yes 

17 

34-69 

5 

29.4 

2 

11.8 

3 

17.6 

4 

23-5 

3 

17.6 


Sophie 

Multiplist 

Yes 

15 

30.61 

2 

13-3 

6 

40.0 

3 

20.0 

1 

6-7 

3 

20.0 

Group 6 

Total 

Nienke 

Evaluativist 

No 

23 

34-33 

9 

39-1 

4 

17.4 

5 

21.7 

4 

17.4 

1 

4-3 

messages: 

67 

Steven 

Evaluativist 

Yes 

9 

13-43 

2 

22.2 

6 

66.7 

0 

0.0 

1 

11.1 

0 

0.0 


Anouk 

Evaluativist 

Yes 

11 

16.42 

7 

63-6 

0 

0.0 

0 

0.0 

2 

18.2 

2 

18.2 
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Number 

of 
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proportion 


Types of contribution 





messages 

in group 
(%) 

Externalizat 

ion 

Elicitation 

Agreemen 

t 

Integratio 

n 

Disagreeme 

nt 






N % 

N % 

N % 

N % 

N % 

Jessy 

Multiplist 

Yes 

24 

35-82 

6 25.0 

5 20.8 

7 29.2 

5 20.8 

1 4.2 
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A check was performed on students’ attitudinal change on the GMO issue from pretest to posttest. Figure 2 
depicts these results, showing for each student the Likert value they chose, pretest and posttest, on the 
question of development of GMOs for the market to improve sustainability. It shows that 14 of 25 (56%) 
students shifted their opinion on the GMO issue, from pretest to posttest. In other words, for these students, 
there was an indication that the dialogue and argumentation affected their attitude to GMOs. 

Of the fourteen students who shifted their opinions on the controversial issue from pretest to posttest, nine 
of these students were multiplists and five were evaluativists, giving the same split (56%) within both these 
subgroups as the whole group. 

The frequency of types of discourse messages was examined between these subgroups, looking at the number 
of messages in each categoiy (externalisation, elicitation etc.), to see if there was any systematic variation or 
if epistemic beliefs were independent of discourse activities (null hypothesis). No significant differences in 
types of discourse messages were found between those students categorised as multiplist and those 
categorised as evaluativist, x 2 (4, N = 25) = 0.14, p = 0.54. A further test splitting multiplists into those that 
shifted attitude on GMOs and those that did not, and evaluativists in the same way, also did not reveal 
significant variations in discourse activity by type of message, x 2 (i2, N = 25) = 1.08, p = 0.24. The largest 
variation in discourse activity between the two categories was that multiplists posted more agreement 
messages and evaluativists posted fewer agreement messages. Multiplists who did not shift their attitude 
asked fewer questions (elicitations) and posted more agreement compared with multiplists who did shift 
their attitude. Evaluativists who shifted their attitude posted much fewer agreements than those evaluativists 
who did not shift their attitude. 
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Figure 2. Attitude shift of student opinions from pretest to posttest in terms of development of GMOs for the 
market to improve sustainability. The letter S represents student names. 

A positive response for students’ satisfaction with the learning experience and its outcomes was obtained 
during the plenary discussion sessions at the end of the experiment. During the plenary discussion sessions, 
students appreciated the game with regard to its dynamic nature, user-friendliness, and variation of the 
sentence openers. Furthermore, they said that the game was useful with respect to practicing, provoking, and 
promoting their critical reasoning and argumentation skills. 

Qualitative analysis of the data for each learning group is presented below in order to have a clear picture of 
the relationship between argumentative discourse activities and students’ epistemic beliefs. Overall, the 
proportions of the type of contributions to the game were highest for externalization (32%), and then 
agreement (21%), elicitation (17%) and integration (16%), with the least belonging to disagreement (14%). 

Group 1 consisted of three multiplists and one evaluativist. Students in this group produced 61 messages, of 
which, 14 messages (23%) were categorized as externalization, 11 messages (18%) as elicitation, eight 
messages (13%) as agreement, 16 messages (26%) as integration, and 12 messages (20%) as disagreement. In 
this group, the opinion of two students on the controversial issue shifted from pretest to posttest. 
Characteristics of each student in this group in terms of epistemic orientation, attitudinal change, number of 
messages per individuals and groups, and type of contribution based on argumentative discourse activities 
are presented below. 

Group 2 consisted of three multiplists, and one evaluativist. Students in this group produced 68 messages, of 
which, 24 messages (35%) were categorized as externalization, 14 messages (21%) as elicitation, 13 messages 
(19%) as agreement, 10 messages (15%) as integration, and seven messages (10%) as disagreement. In this 
group, the opinion of only one student on the controversial issue shifted from pretest to posttest. 

Group 3 is rather different from most other groups in having three evaluativists and only one multiplist. 
Students in this group produced 68 messages, of which 30 were categorized as externalization (44%). Only 
seven messages (10%) were categorized as elicitation, and another three messages (4%) were categorized as 
agreement. Of the remaining 28 messages, seven were classified as integration (10%) and 21 as disagreement 
(31%). In this group, the opinion of three students on the controversial issue shifted from pretest to posttest. 

Group 4 consisted of four multiplists and one evaluativist. Students in this group produced 90 messages, of 
which 20 messages (22%) were categorized as externalization, 27 messages (30%) as elicitation, 23 messages 
(26%) as agreement, 14 messages (16%) as integration, and only six messages (7%) as disagreement. In this 
group, the opinion of three students on the controversial issue shifted from pretest to posttest. 

Group 5 is rather different from other groups in terms of the distribution of students on their epistemic 
orientation, since it consists of only multiplists. Students in this group produced 49 messages, of which, 17 
messages (35%) were categorized as externalization, nine messages (18%) as elicitation, nine messages (18%) 
as agreement, seven messages (14%) as integration, and seven messages (14%) as disagreement. In this 
group, the opinion of two students on the controversial issue shifted from pretest to posttest. 

Group 6 consists of three evaluativists and one multiplist. Students in this group produced 67 messages, of 
which 24 were categorized as externalization (36%), and 15 messages (22%) as elicitation. The same number 
of messages have been classified as agreement and integration: 12 messages (18%). A mere four messages 
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(6%) were classified as disagreement. In this group, the opinion of three students on the controversial issue 
shifted from pretest to posttest. 


Figure 3 shows a typical fragment of the discussion from Group 1 demonstrating the nature of their 
interactions. This particular fragment was chosen because it includes disagreement. The first part starts with 
Tom eliciting group views on the topic, to which Simon outlines two opposing positions on the nature of 
GMOs (externalisation). Robert requests clarification (elicit.) by challenging the position for GMOs and Tom 
defends by elaborating on the pro-GMO position (ext.). Robert makes a tentative disagreement and Simon 
elaborates (externalisation) on Tom's position. In the second part, Robert agrees with, and elaborates 
(integration) on Simon's first statement on the unnaturalness of GMOs. Tom disagrees with Robert and 
restates both their respective positions. In turn, Simon disagrees with Tom and criticises his position. 
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GMOs have been touted as both nightmare and salvation—a nightmare to human health and 
biodiversity, and salvation for human food production, reducing pesticide emissions to the 
environment, and, by extension, biodiversity and human health. 

Tom 

What do you think? What do you think of GMOs in respect of interfering with natural selection and 
evolution processes?? 

Simon 

I think that it feels very unnatural to interfere with nature, but in nature a mutation can also occur. 
Some say evolution is just sped up thanks to GMO research. 

Robert 

Are you saying that we should use GMOs as we will? Using the technoogy to eventually alter our 
own selves? 

Tom 

Let me say more about that. I don't want to say that we should use GMOs as we will. But 
what I would argue is that GMOs are a part of our abilities that we have gained through 
evolution/natural selection and that therefore it is "natural" for humans to use GMO 
technologies 

Robert 

I'm not so sure . Isn't technology in itself unnatural? 

Simon 

I'm not so sure . There are people who argue that nature is what WE create. This is a 
very difficult discussion and it does not assume that there is one singe truth but that 
reailty is cnstructed. In that sense technology could be a NEW nature. (BTW excuse 
my spelling error, my keyboard is not functioning a 100%). 

Robert 

I agree because what do we really know about the incredible complex planet, we call earth? 
We've only been around for several what, thousands of years? Most of us don't even know who 
we are ourselves, let alone what other species are and how they might react. 

Tom 

I disagree because . What I imply from your response is that you think Humans and nature are 
not the same thing. And that what we as humans are doing with the earth is not part of the 
earth. What I think is that humans are part of this earth and that we are doing with it is the 
result of natural processes. 

Simon 

I disagree because I do not think that tinkering with nature and changing it on such a 
fundamental level is a natural process. Just because we found out how to do that does not 
mean that it is natural. I do not think that we as human can position ourselves above nature 
in the sense that we can do with it whatever we want. 


Figure 3. Fragment of Group 1 GMO discussion demonstrating disagreements. 


Discussion 

The discussion pattern reveals the style of discussion throughout, in that students used good reasoning and 
elaborated well, and were not afraid to use disagreement, if needed, to challenge their peers. The dialogue 
game provided guidance and shape to the students' choice of responses through the selection of openers and 
encouraged explicit and elaborated statements (e.g. “Let me say more about that...”). It also encouraged them 
to commit to disagreement (“I disagree because...”) or provide evidence (“I read that...”) through the use of 
the suggested openers, whereas most students in other contexts, are normally circumspect with regard to 
these dialogue moves. Previous work on InterLoc has shown that the dialogue game significantly changes the 
style of discussion and the frequency of the lesser used dialogue moves, compared with a non-guiding 
environment (chat). 
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The results show little effect of epistemic beliefs on the style and frequency of particular types of contribution 
by students. It was expected that multiplists would interact less and be less critical of their peers than 
evaluativists. Also that multiplists would be less susceptible to attitude shifts as a consequence. However, this 
did not appear to be the case in the present study and so a number of factors were considered to see why this 
might have been so. The study of Kuhn et al. (2000) tested epistemic beliefs in several domains, including 
social truths and physical truths, and across age and experience ranges, including undergraduates. Assuming 
the GMO issue in this study falls into both social and physical truths domains, the average across these two 
domains for the Kuhn study for undergraduates was as follows: evaluativist (42.5%), multiplist (47.5%), and 
absolutist (5%). The proportions for this study (all undergraduates) were not dissimilar, and were as follows: 
evaluativist (36%), multiplist (64%), and absolutist (0%). Fourteen (56%) students shifted their opinion (as 
measured by the Likert scale) on the GMO issue, between the pretest and posttest on the two attitude 
statements (“GMOs should be further developed for the market to improve sustainability” and “GMOs are a 
danger to biodiversity”). The first issue was the subject for the main debate between the students, and this 
showed the most shifts in opinion with seven students responding more positively after the debate and four 
students more negatively. Each opinion shift recorded represents a clear revision by the student. For 
instance, two students moved from a neutral position to being against further development, and three 
students from being neutral to supporting development. However, we believe another factor that may be 
involved in change of opinion is the strength or weakness of prior knowledge on the controversial issue. If 
prior knowledge is weak and the pretest opinion is not firmly held, an opinion may be changed with relatively 
little argument. 

The proportion of multiplists and evaluativists shifting opinion was the same at 56%. There were six students 
who strongly favoured further development of GMOs at the outset and who did not change their opinion. 
Mainly, they did not regard GMOs as a threat to biodiversity. Looking for differences between evaluativists 
and multiplists in the discussion, it appears that there were few differences, and none statistically significant. 
Evaluativists produced more messages, with fewer agreement and more disagreement messages, while 
multiplists produced fewer integration messages. This study did not replicate the findings of Nussbaum et al. 
(2008), that multiplists are less critical regarding inconsistencies and misconceptions and less interactive 
with their partners than other belief groups. The percentage of multiplist messages categorised as 
disagreement (12.9%) was slightly less than the average (14.1%), and the percentage of externalizations 
(32.3%) slightly less than the average (34.5%). Since externalizations are viewed as the least interactive 
category, multiplists in this study appeared to interact about as much as the others. 

Previous studies in this field had found differences in the style and strength of interactions within the 
discussion emerging from the differences in epistemic beliefs, but this study found none. Having followed the 
methodology from the Kuhn study (Kuhn et al., 2000), there are only a few differences in context that could 
have caused these results. One background difference (which seems to the authors to be unlikely) is culture, 
since this study was carried out in The Netherlands. There are three likely remaining differences, and each 
affects the willingness to argue with peers: the use of a controversial issue for discussion; the context of the 
discussion within a university course; and, the use of a dialogue game. The controversial issue of GMOs 
caused quite passionate views on both sides of the argument, increasing the willingness to argue. We can see 
the strong views expressed by many of the students on the controversial issue, indicated by their positions 
towards the extremes on the Likert scales in Figure 2 and, for example, the number of disagreements in the 
discussion fragment in Figure 3. The context of the discussion, within a university course the students had 
chosen, confirmed their personal interest in the topic and would be expected to raise their willingness to 
argue. Finally, the use of a dialogue game which promotes and scaffolds critical reasoning and argument and 
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enabled the students to engage with their peers without recourse to, or fear of, personal (ad hominum) 
statements, increasing willingness to argue. 

While we can make no strong claims from a small number study such as this, the indications are that the 
activity design is responsible for the enhanced willingness to argue and engage with others for all students, 
eliminating the expected interaction differences due to multiplist or evaluativist beliefs. In addition, the 
results showed the same percentage of students making attitude shifts between the two groups, indicating an 
equal openness to persuasion. The change between being neutral to supporting a proposal about developing 
GMOs is in fact a relatively large change for a student studying the topic, so none of the attitude shifts 
recorded were trivial or unconsidered to the students involved. This study did not test conceptual change, but 
the equal frequency of attitude shifts, which indicate openness to persuasion, are at least likely to have 
implications for frequency of conceptual change also between the groups. 

Implementation of the digital dialogue game for debating a controversial issue was evaluated positively by 
undergraduate students as can be seen in their satisfaction with the learning experience and its outcomes. 
This was documented through the plenary discussion session at the end of the game. This was apparently a 
challenging, yet pleasurable, learning opportunity for undergraduate students to debate with fellow students 
over GMOs—one of the most controversial issues of the social and physical sciences. The use of sentence 
openers guided the students to elicit information, ask clarifying questions, express agreements and 
disagreements, and integrate various points of view. The dialogue game provided a safe and respectful 
environment for students to practice their argumentation and exercise critical discussion and reasoning 
skills. It was also easy for students to quickly learn various functionalities of this game-based environment. 
Taking all together, the user-friendliness and the design of the sentence openers of the game offered through 
InterLoc were positively reflected in the learners’ scores for satisfaction with the learning experience. 


Conclusions, Limitations, and Suggestions for Future Research 

Within this study a learning activity was designed to positively engage students by (a) choosing a 
controversial topic, (b) creating groups that contain conflicting opinions, and (c) using a dialogue game to 
direct the conflict towards deeper reasoning and engagement. The results showed that more than half of the 
students shifted their opinions in the questionnaire after the debate, and the transcripts demonstrated that 
they had engaged deeply with their peers. The measurement of different epistemic beliefs among the 
students led to expected differences in style of argumentation and level of interaction between students with 
multiplist beliefs and those with evaluativist beliefs. No significant differences between the two groups were 
found in this study, leading to a re-evaluation of the relationship between students’ argumentative discourse 
activities, epistemic beliefs, and change in attitude when exposing them to an activity design with enhanced 
motivational characteristics. Based on this study, the indications are that use of an appropriate learning 
design, such as a digital dialogue game and an appropriate choice of controversial issue, enhanced the 
willingness to argue of all students, whatever a student's epistemic beliefs. It is also extremely likely that 
selection of group members to include opposite views played a part in increasing willingness to argue. Many 
variables are involved in such engagements, not all under the learning designer's control, but the activity 
design and the authenticity of the engagement are undoubtedly the two major factors contributing to 
willingness to argue. This is good news for learning designers who may be able to arrange suitably engaging 
and thoughtful argumentative discourse, despite students' epistemic beliefs and reluctance to engage. The 
argumentative discourse in this study caused most students to change their position and shift their opinion, 
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an outward sign that the activity initiated thinking, and rethinking, among the students. Of course, further 
and more detailed research is needed to confirm these indications, and determine if epistemic beliefs have 
more subtle effects on students’ participation in such debates as were undertaken here. 

One of the strengths of the present study is that the activity took place in an educational setting with high 
ecological validity, and not in an artificial setting which might reduce student motivation. Our criticism of 
previous studies in this area would be that lack of authentic motivation might dramatically affect the results 
obtained (with regard to the lack of engagement of multiplists for instance), yet in real-life situations where 
the issues matter to the students, the multiplists may argue equally as hard as evaluativists. This study 
provided the opportunity to shed light on the effects of such a designed game-based activity on 
argumentative discourse and attitudinal change as they occur in authentic learning situations. In educational 
settings, not only are students’ self-reported opinions on the learning important, but also their actual 
learning and achievement. Therefore, we might aim to incorporate any learning metrics, from course exams 
or assignments, to measure actual learners’ achievement in educational settings with such interventions. 

This study did not count for students’ prior knowledge on GMOs. As discussed previously, prior knowledge 
might play a role in shifting perceptions on the controversial issues. When prior knowledge is strong then 
often opinion is firmly held and would not be easily changed or shifted compared with a weak prior 
knowledge that can be changed with relatively little argument. Thus, future research should focus on taking 
prior knowledge of players into consideration. This study did not experiment with group size, and so future 
work should be scaled up in order to test alternative configurations of group size and conflicting opinions 
within a group. Finally, with further research we will be able to draw more certain conclusions about the 
likely impact of epistemic beliefs on willingness to argue and the likelihood of rethinking and opinion shift in 
students, and whether, as this study indicates, these are in fact variables in the control of the learning 
designer. 

Such debates as were undertaken here, are part of a learning designer's goal to facilitate thoughtful and 
deeper argumentation which in turn facilitate students' grasp of what it means to be a 'critical thinker'. We 
conclude that even this small scale study reminds us of the many variables at work within a learning design 
affecting willingness to argue and engagement in argumentative discourse. They include the ecological 
validity of the setting, knowledge and pertinence of the issue at hand, whether there is confrontation of 
conflicting views, students' epistemic beliefs, and, by no means least, the style of engagement (in this case an 
online dialogue game). Outcomes are not determined by one variable alone, so learning designers will need 
to keep in mind the full range of factors that will facilitate thoughtful and deeper argumentation. 
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